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CHAPTER  ONE 
INTRODUCTION 

The  Massachusetts  Educational  Assessment  Program  (MEAP),  which  you 
administered  last  spring,  is  different  from  the  testing  programs  that  schools  usually 
administer  to  their  students.  In  order  to  understand  how  the  assessment  results 
can  be  useful,  it  is  important  to  understand  the  ways  in  which  these  programs 
differ  and  why. 

Standardized  achievement  tests  are  not  designed  to  measure  the  effectiveness 
of  your  instructional  program  or  the  specific  learning  problems  of  individual 
children.  They  are  designed  to  give  a  general  indication  of  the  performance  of 
individual  students.  They  accomplish  this  by  reporting  performance  in  relation 
to  some  established  norm. 

This  purpose  determines  both  the  content  of  the  test  and  the  way  items  are 
chosen.  For  example,  because  standardized  achievement  tests  are  designed  to  be 
used  in  many  different  types  of  schools  throughout  the  country,  with  many 
different  instructional  programs,  their  content  must  reflect  the  basic  knowledge 
and  skills  that  are  common  to  the  curriculum  of  all  schools.  Secondly,  because 
standardized  tests  must  give  reliable  results  for  individual  students,  the  statistical 
character  of  test  questions  is  of  paramount  importance  in  their  selection.  Items 
are  chosen  on  the  basis  of  their  power  to  discriminate  among  students.  They  must 
be  highly  correlated  and  of  moderate  difficulty.  Items  that  may  be  of  interest  in 
terms  of  their  content  are  eliminated  if  they  do  not  conform  to  these  statistical 
expectations. 

Assessment  tests,  such  as  the  Massachusetts  Educational  Assessment  Program 
tests,  have  a  different  purpose.  They  are  not  designed  to  give  information  about 
individual  students;  they  are  designed  to  give  information  about  the  effectiveness 
of  the  curriculum.  They  do  not  measure  what  is  taught  at  a  particular  grade  level 
or  is  commonly  taught  among  districts,  but  they  cover  as  much  as  possible  of 
the  subject  area  domains  that  students  might  be  expected  to  learn  up  to  the  grade 
level  tested.  To  accomplish  this  purpose,  their  content  must  be  much  broader  than 
that  of  standardized  achievement  tests,  encompassing  the  many  different  kinds 
of  knowledge  and  thinking  processes  that  reflect  the  content  area.  In  order  to 
provide  this  broad  coverage,  large  numbers  of  test  questions  must  be  used.  In 
the  case  of  the  MEAP,  these  questions  are  distributed  across  many  different  test 
forms,  with  each  student  completing  only  one  form  of  the  entire  test. 

Aspects  of  test  design,  such  as  item  statistics  which  determine  the  acceptability 
of  items  on  standardized  tests,  are  of  less  concern  in  an  assessment.  Because  the 
results  are  reported  in  terms  of  a  collection  of  students,  individual  differences  in 
student  responses  tend  to  cancel  each  other  out.  For  this  reason,  the  average 
obtained  over  many  different  kinds  of  items  by  a  group  of  students  tends  to  be 
highly  reliable.  In  other  words,  the  test  results  of  a  group  of  students  who  have 
been  exposed  to  a  curriculum  is  considered  a  good  indication  of  the  effectiveness 
of  that  curriculum. 

1 


Because,  at  a  state-wide  level,  it  is  clearly  impossible  to  reflect  the  individuality 
of  all  the  schools  in  the  Commonwealth,  this  booklet  has  been  designed  to  help 
you  use  the  results  of  the  assessment  to  evaluate  your  own  curriculum.  The 
assessment  results  are  reported  in  a  variety  of  ways.  Some  may  be  more  helpful 
to  you  than  others.  In  this  booklet,  we  explain  the  various  ways  in  which  types 
of  information  have  been  reported  and  suggest  ways  that  you  might  use  them 
in  program  evaluation. 

Chapter  Two  provides  background  on  the  assessment  program,  discussing  its 
intent  and  how  that  purpose  has  been  carried  out.  In  Chapter  Three,  we  review 
the  various  scores  in  the  report,  demonstrating  the  kinds  of  comparisons  that  you 
can  make.  In  Chapter  Four,  we  suggest  a  process  by  which  you  can  use  the  results 
to  examine  the  effectiveness  of  your  own  program.  Finally,  in  Chapter  Five,  we 
give  some  examples  of  schools  that  used  the  1986  assessment  results  as  a  means 
of  program  improvement. 

The  purpose  of  the  assessment  is  to  provide  information  to  schools,  information 
that  is  often  costly  or  impossible  to  obtain  within  the  system  itself.  The  process 
should  not  stop  there,  however.  In  a  certain  sense,  the  assessment  reports  which 
you  have  received  should  represent  only  a  beginning.  They  should  be  used  not 
to  give  answers  but  to  generate  questions.  The  purpose  of  this  handbook  is  to 
help  you  understand  the  kinds  of  questions  that  can  be  asked  and  the  ways  in 
which  answers  can  be  found. 


CHAPTER  TWO 
BACKGROUND:     THE  MASSACHUSETTS  EDUCATIONAL 

ASSESSMENT  PROGRAM 

The  Massachusetts  Educational  Assessment  Program  was  established  in  1985 
as  one  of  a  number  of  programs  designed  to  improve  curriculum  and  instruction 
in  the  public  schools  in  Massachusetts.  Under  the  mandate  by  which  it  was 
established,  it  is  designed  with  two  purposes: 

•  to  provide  comparisons  of  peformance  at  school,  district,  state  and  national 
levels,  when  possible; 

•  to  improve  curriculum  and  instruction. 

In  1985-86,  the  first  year  of  the  assessment,  third,  seventh  and  eleventh  grade 
students  were  tested  in  reading,  mathematics  and  science.  In  the  spring  of  1988 
the  grades  at  which  students  were  tested  were  changed  to  fourth,  eighth  and  twelfth 
grades,  and  social  studies  was  added  to  the  subject  areas  tested.  These  grade  levels 
and  subject  areas  were  consistent  with  those  targeted  by  the  National  Assessment 
of  Educational  Progress  (NAEP)  so  that  national  comparisons  can  be  made. 


Test  Development 

The  objectives  which  form  the  framework  for  each  content  area  are  described 
in  a  separate  publication,  Description  of  Test  Content  and  Reporting  Categories.  These 
objectives  are  arrived  at  through  an  iterative  process  that  involves  national 
curriculum  experts,  state  curriculum  experts,  the  Department  of  Education  and  the 
schools  themselves. 

Initially,  the  Curriculum  Advisory  Committees,  composed  of  teachers  and 
curriculum  coordinators  throughout  the  state,  review  the  set  of  objectives  drawn 
up  by  the  National  Assessment  of  Educational  Progress,  as  well  as  those  from 
other  sources.  The  Committees  use  this  material  to  construct  a  tentative  objectives 
framework  for  each  subject,  including  the  major  content  areas  and  types  of  thinking 
that  are  considered  to  be  the  essential  components  for  achievement  within  that 
discipline.  They  also  draw  up  a  list  of  content  topics  for  consideration  by  schools. 
All  schools  are  asked  to  react  to  this  list  by  rating  each  topic  in  terms  of  its 
importance  for  student  learning  by  the  target  grade  level  (4,  8,  12).  Committees 
then  use  the  results  of  this  survey  to  finalize  the  objectives  framework  and  to  decide 
what  specific  aspects  of  knowledge  or  skills  should  be  included  within  each  specific 
subcategory  of  the  framework.  Test  questions  are  then  drawn  from  the  National 
Assessment  of  Educational  Progress  (NAEP)  and  other  sources,  or  are  newly  written 
in  order  to  reflect  the  stated  objectives.  The  committees  contribute,  review  and 
approve  all  items  before  they  are  used  in  the  tests. 
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Other  Information 

In  addition  to  the  cognitive  tests,  a  series  of  student,  teacher  and  principal 
questionnaires  are  also  developed  using  a  similar  process.  Many  of  the  questions 
are  identical  to  those  used  by  NAEP  and  other  national  programs;  others  are 
designed  by  the  Department,  with  the  advice  of  the  Curriculum  Advisory 
Committees.  Responses  to  the  student  questionnaires  are  primarily  used  to  interpret 
test  results.  They  form  the  basis  for  the  Comparison  Score  Bands  and  are  included 
in  the  school  reports  as  additional  information  to  schools.  Responses  to  the  teacher 
and  principal  questionnaires  are  aggregated  and  reported  at  the  state  level.  Their 
primary  use  is  in  the  exploration  of  policy  and  curriculum  issues;  however,  they 
are  also  used  as  indicators  of  the  overall  quality  of  education  in  Massachusetts, 
particularly  in  terms  of  national  comparisons. 


CHAPTER  THREE 
MAKING  COMPARISONS 

Schools  have  many  ways  to  chart  the  progress  of  individual  students:  teacher 
observations,  classroom  tests,  homework  assignments,  student  projects,  and 
standardized  tests.  However,  it  is  far  more  difficult  for  a  school  or  district  to  step 
back  and  chart  its  own  course.  When  schools  ask  "How  effective  is  our  program? 
Are  we  doing  the  best  job  possible?",  there  are  few  tools  at  hand  for  judging  their 
progress. 

One  way  to  measure  a  school's  achievement  is  to  compare  the  total 
performance  scores  of  its  students  with  those  of  students  in  other  schools  or  the 
state  as  a  whole.  Another  way  is  to  look  at  the  performance  of  students  in  similar 
schools.  Another  is  to  look  internally,  at  performance  among  groups  of  students 
within  the  school.  The  school  report  which  you  have  received  gives  information 
on  all  these  ways,  and  more.  The  purpose  of  this  chapter  is  to  demonstrate  the 
kinds  of  comparisons  you  can  make  and  to  help  you  in  interpreting  their  meaning. 


Overall  Comparisons 

In  looking  at  your  report,  the  first  and  most  obvious  questions  will  probably 
be:  How  does  our  student  performance  compare  with  some  sort  of  standard?  On  the 
average,  are  we  performing  better  or  worse?  There  are  different  ways  to  answer  these 
questions,  depending  upon  the  standard  chosen. 

1.     Comparisons  with  your  district  and  the  state 

On  page  5  of  your  school  report  you  will  find  the  scaled  scores  which  your 
students  achieved  in  each  of  the  four  subject  areas  tested.  Scores  are  given  not 
only  for  the  entire  content  area  but  for  each  of  the  subcategories  that  appeared 
on  the  test.  Your  score  in  these  content  areas  will  tell  you  how  students  in  your 
school  performed  on  these  tests  in  comparison  to  your  district  and  to  the  state 
as  a  whole. 

In  order  to  make  meaningful  comparisons  among  schools,  across  subject  areas, 
and  between  years,  these  scores  are  scaled.  That  is,  the  percentage  of  correct 
responses  is  transformed  to  a  common  measure,  which  is  based  on  the  normal 
distribution.  The  mean  state  score  is  set  at  1300,  with  68%  of  school  scores  lying 
between  1200  and  1400.  Sixteen  percent  of  the  scores  are  below  1200,  and  another 
16%  are  above  1400.  Because  the  standard  deviation  is  set  at  100  points,  scores 
that  differ  by  less  that  50  points  should  not  be  considered  substantially  different 
or  educationally  meaningful. 
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2.     Comparisons  with  similar  schools 

On  page  5  you  will  also  find  your  Comparison  Score  Band,  which  will  help 
you  compare  the  scores  of  your  students  with  those  of  other  students  in  schools 
like  yours.  Comparison  Score  Bands  are  determined  by  relating  certain  student 
background  factors  (such  as  parental  education,  English  fluency,  length  of  residence 
in  the  United  States)  and  certain  community  factors  (such  as  percentage  of  students 
receiving  free  or  reduced  price  lunch  and  Kind  of  Community)  to  performance 
on  the  assessment.  The  relative  influence  of  each  factor  on  performance  is 
determined  for  the  state  as  a  whole,  and  the  resulting  formula  is  used  to  compute 
the  Comparison  Score  Band  for  each  individual  school  (See  page  7  of  your  report 
for  a  comparison  of  the  values  of  those  background  factors  between  your  school 
and  the  state  and  district.)  Since  all  schools  differ  with  respect  to  the  factors 
involved,  each  school  has  its  unique  Comparison  Score  Band.  Its  width  reflects 
its  reliability  and  is  determined  in  large  part  by  the  number  of  students  in  the 
school. 

Comparison  Score  Bands  are  an  important  reference  point.  They  acknowledge 
the  fact  that  factors  beyond  the  control  of  schools  affect  students'  academic 
performance.  However,  it  is  important  to  remember  that  they  are  not  prescriptive; 
they  are  normative.  In  other  words,  they  do  not  indicate  where  a  school  should 
score,  but  only  where  it  might  be  expected  to  score  given  the  background  factors 
involved.  On  the  other  hand,  scores  that  fall  below  the  band  should  be  examined 
carefully.  They  indicate  that,  in  those  areas,  your  students  performed  less  well 
than  other  students  with  similar  economic  and  social  backgrounds. 


In  a  separate  document,  the  average  scores  for  schools  in  each  Kind  of 
Community  (KOC)  are  given.  Schools  are  grouped  into  KOCs  according  to  different 
economic  factors  in  their  communities.  Unlike  Comparison  Score  Bands,  KOCs  do 
not  reflect  the  background  factors  of  the  actual  students  who  took  the  test  and 
are,  therefore,  grosser  measures  of  similarity.  Their  advantage  lies  in  the  fact  that 
communities  within  each  KOC  are  named  and  described.  On  the  other  hand,  a 
list  of  schools  or  districts  with  Comparison  Score  Bands  similar  to  yours  is  available 
from  the  Department  should  you  desire  it. 


3.    Comparisons  with  previous  assessment  results 
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On  page  8  of  your  report  we  give  the  results  of  the  first  assessment  in  1986. 
Although  they  are  not  directly  comparable  to  this  year's  results,  they  can  be  used 
in  your  evaluation. 

Since  the  purpose  of  the  assessment  is  to  measure  the  effectiveness  of  your 
curriculum,  it  is  designed  to  reflect  the  cumulative  process  of  education.  In  other 
words,  the  performance  of  your  students  at  grade  4  is  the  result  not  only  of  their 
instruction  at  grade  4,  but  of  their  previous  schooling.  For  this  reason,  consistent 
areas  of  strengths  or  weaknesses  within  the  two  grades  at  any  of  the  levels  tested 
in  the  two  years  is  stronger  evidence  than  can  be  offered  by  either.  When  results 
converge,  they  have  increased  reliability.  On  the  other  hand,  because  learning  is 
cumulative,  recent  program  changes  may  not  be  immediately  apparent  in  these 
overall  results.  The  specific  curriculum  area  results  found  on  pages  10  to  13  of 
your  report  are  probably  a  better  indicator  of  program  effectiveness. 
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Internal  Comparisons 

Although  overall  comparisons  are  a  good  way  to  start  the  process,  they  are 
just  that — comparisons  in  terms  of  a  total  scaled  score.  To  use  the  assessment 
results  effectively,  they  must  be  examined  in  terms  of  your  own  students  and  your 
own  curriculum.  The  fact  that  a  school  has  performed  well  in  comparison  to  the 
state  or  district  score  gives  very  little  useful  information  in  itself.  Performance 
must  be  measured  in  terms  of  the  school's  own  goals  and  objectives. 

1.     Comparisons  among  groups  of  students 

The  distribution  of  scores  in  your  school  gives  an  indication  of  how  well  you 
are  serving  different  groups  of  students.  This  information  is  given  on  page  7,  where 
the  number  of  your  students  in  each  quartile  is  compared  with  the  state  and  district. 


Student  Score  Distribution 

Content  Area 

Quarter 

Students  in  Each  Quarter  of  the 
State  Student  Distribution 

State 

District 

School 

% 

N 

% 

N 

% 

Reading 

Highest 
Third 
Second 
Lowest 

25% 
25% 
25% 
25% 

40 
40 
50 
50 

22% 
22% 
28% 
28% 

22 
20 
17 
17 

24% 
22% 
19% 
19% 

Mathematics 

Highest 
Third 
Second 
Lowest 

25% 
25% 
25% 
25% 

40 
45 
45 
50 

22% 
25% 
25% 
28% 

5 

11 
44 
30 

6% 
12% 
49% 
33% 

Science 

Highest 
Third 
Second 
Lowest 

25% 
25% 
25% 
25% 

30 
60 
60 
30 

17% 
33% 
33% 
17% 

11 
21 
30 
28 

12% 
23% 
33% 
31% 

Social  Studies 

Highest 
Third 
Second 
Lowest 

25% 
25% 
25% 
25% 

45 
45 
50 
40 

25% 
25% 
28% 
22% 

12 
40 
21 
17 

13% 
44% 
23% 
19% 

The  distribution  of  student  scores  will  give  a  profile  of  your  school's 
performance.  It  will  help  in  your  interpretation  of  total  subject  scores  and  may 
help  you  to  identify  groups  of  students  who  are  atypical.  For  example,  schools 
with  the  same  overall  score  can  have  very  different  distributions  of  students  in 
each  quartile,  which  will  indicate  different  areas  to  be  examined. 


Quartiles 

State: 
Average  Score:  1300 

School  A: 
Average  Score  :  1300 

School  B: 
Average  Score:  1300 

Highest 
Third 
Second 
Lowest 

25 
25 
25 
25 

15 
43 
27 
15 

31 

17 
37 
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Although  the  majority  of  schools  that  score  at  or  close  to  the  state  average 
of  1300  generally  follow  the  expected  distribution,  this  is  not  necessarily  the  case. 
For  example,  contrast  the  profiles  of  School  A  and  School  B,  both  of  which  received 
close  to  1300  in  the  1986  assessment  of  Reading. 

The  distribution  of  scores  for  School  A  suggests  a  fairly  homogeneous  group 
of  students.  Over  70%  of  student  scores  are  within  the  second  and  third  quartile 
(in  contrast  to  50%  of  the  state  as  a  whole).  In  fact,  with  58%  scoring  at  or  above 
the  state  average,  the  school  score  of  1300  slightly  underestimates  the  scores  of 
its  students. 

In  contrast,  the  distribution  of  scores  for  School  B  highlights  the  difference 
in  performance  between  two  groups  within  the  same  school.  The  number  of 
students  at  each  extreme  is  sizable  and  raises  questions  about  the  instructional 
programs  that  have  been  developed  for  their  differing  needs.  Whereas  the  score 
of  1300  may  suggest  that  performance  in  this  school  is  "average,"  the  distribution 
indicates  that  this  is  not  the  case. 


School  C: 

School  D: 

Quartiles 

Average  Score  :  1460 

Average  Score:  1150 

Highest 

50 

15 

Third 

18 

8 

Second 

12 

18 

Lowest 

20 

59 

Although  the  two  schools  above  appear  to  perform  quite  differently  in  terms 
of  their  total  scores,  in  both  cases  a  substantial  proportion  of  students  performed 
at  a  level  that  was  not  reflected  in  that  score.  They  illustrate  the  fact  that  an 
average  masks  the  variation  in  scores  that  can  occur  within  a  school  population. 
Whatever  a  school's  total  score  may  be,  usually  a  number  of  students  perform 
significantly  better  or  worse  than  that  score  would  indicate.  For  example,  although 
the  average  score  for  School  C  would  indicate  a  very  high  performance  level  when 
compared  with  the  state  as  a  whole,  fully  20%  of  their  students,  or  1  in  5,  are 
performing  at  the  lowest  end  of  the  scale.  Conversely,  15%  of  the  students  in 
School  D,  whose  average  score  was  at  the  lower  end  of  the  state  distribution, 
performed  very  well. 

The  purpose  of  these  examples  is  to  illustrate  the  need  to  examine  the 
distribution  of  scores  within  your  school  carefully.  When  an  atypical  number  of 
students  are  scoring  in  the  lower  quartiles  of  the  distribution,  questions  that 
administrators  might  ask  are: 

•  Is  this  pattern  reflected  across  all  the  subject  areas  or  does  it  reflect  a  particular 
handicap,  such  as  lack  of  language  proficiency? 

•  Is  this  pattern  confirmed  by  results  in  the  Basic  Skills  Testing  Program?  Have 
the  low  scoring  students  been  identified? 


•  How  successful  is  our  remedial  program?  Are  we  stressing  only  low  level  skills 
in  a  single  content  area  to  the  detriment  of  knowledge  in  other  areas? 

•  Are  these  students  receiving  the  proper  support?  Challenges? 

Conversely,  when  the  distribution  shows  an  atypical  number  of  students  in  the 
upper  quartiles,  administrators  might  ask: 

•  Is  this  distribution  reflected  in  scores  on  other  tests? 

•  Have  these  high  scoring  students  been  identified? 

•  Do  their  scores  reflect  the  results  of  a  particular  program  (acceleration,  Gifted  and 
Talented,  Advanced  Placement)?  If  not,  should  there  be  a  program  for  these 
students? 


2.     Comparisons    among    students    in    terms    of    course    work    and    program 
participation 

A  final  comparison  can  be  made  in  terms  of  course  work  and  program 
participation.  This  information  can  be  found  on  pages  7  and  15  of  the  school 
report. 

In  your  interpretation  of  scores  it  is  important  to  be  aware  of  the  extent  to 
which  scores  are  based  on  the  performance  of  different  types  of  students.  As  you 
know,  some  categories  of  students  who  do  not  participate  fully  in  the  entire  school 
program  are  tested.  Not  surprisingly,  their  scores  are  often  lower  than  those  of 
other  students.  Since  schools  differ  in  the  proportion  of  their  populations  who  have 
special  needs  or  who  lack  proficiency  in  English,  the  relative  influence  of  the  scores 
of  these  students  differs  from  school  to  school. 

When  interpreting  your  results  it  may  be  helpful  to  ask  the  following  questions 
in  regard  to  program  participation: 

•  How  does  the  number  of  students  in  special  programs  (Special  Education  or 
Bilingual)  in  this  school  compare  with  that  of  the  rest  of  the  state  and  the  district? 
Are  we  excluding  a  disproportionate  number  of  those  students  from  the  assessment? 

•  To  what  extent  does  our  total  score  reflect  the  performance  of  students  in  special 
programs  (Special  Education  or  Bilingual)? 

•  How  does  the  performance  of  these  students  compare  with  others  throughout  the 
state?  Based  on  these  results,  are  we  satisfied  with  our  program?  When  we  exclude 
students  who  fall  into  these  categories  from  our  total  score,  how  do  we  compare 
with  other  schools  in  the  state  or  district? 

For  high  school  administrators  and  curriculum  evaluators,  scores  are  also  given 
for  students  in  different  courses.  Although  we  do  not  report  the  appropriate  sub- 
category score  for  these  students,  their  total  subject  scores  are  given  with  reference 
to  those  obtained  by  similar  students  throughout  the  state. 
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The  comparisons  suggested  above  are  merely  different  lenses  through  which 
you  can  view  the  assessment  results.  Taken  together,  they  should  give  you  a 
"feel"  for  what  your  score  reflects.  They  acknowledge  the  fact  that  schools  are 
not  factories,  turning  raw  material  into  finished  products,  but  are  institutions 
devoted  to  children  with  different  problems  and  needs.  Any  achievement  score 
is  the  result  of  the  interaction  between  the  curriculum  and  instruction  that  a  school 
delivers  and  the  children  which  it  services.  It  is  relative,  not  only  numerically  but 
in  terms  of  what  it  represents,  since  it  is  based  on  different  groups  of  students. 
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CHAPTER  FOUR 
DEVELOPING  AN  ACTION  PLAN 

It  is  not  the  purpose  of  this  handbook  to  tell  you  how  to  improve  curriculum 
and  instruction.  There  are  other,  better  sources  for  this.  Instead,  its  purpose  is 
to  help  you  use  the  assessment  reports  to  look  at  your  curriculum  and  instruction 
and  to  ask  the  right  questions.  The  answers  to  these  questions  must  come  from 
you,  with  the  help  of  your  staff  and  other  interested  individuals. 

In  setting  priorities,  it  is  important  to  review  your  entire  program.  Although 
comparisons  with  state  and  district  give  relative  performance,  in  no  sense  do  they 
indicate  how  well  your  students  have  performed  in  any  absolute  sense.  This  is 
because  the  assessment  is  not  a  competency  test;  it  does  not  put  forth  specific 
objectives  which  we  believe  that  all  students  should  master.  Unlike  Basic  Skills 
testing,  there  is  no  absolute  standard  of  knowledge  involved.  As  a  corollary  to 
this,  a  relatively  high  total  score  on  the  assessment  does  not  indicate  a  uniformly 
superior  program.  In  order  to  judge  how  your  program  can  be  improved,  it  is 
necessary  to  look  at  the  specific  curriculum  area  results  for  indications  of  strengths 
and  weaknesses. 


Developing  An  Action  Plan 


Ql 


Q2 


Q3 


Q4 


What  are  the  strengths  &  weaknesses  of  our  instructional  program? 

Compare  specific  objectives  with  overall  content  score 


v 


Are  we  satisfied  that  this  is  an  area  of  weakness? 

Compare  scores  with  those  from  other  sources 


V 


Is  this  topic  or  skill  part  of  our  intended  curriculum? 

Review  written  curriculum,  textbooks,  teaching  materials 


V 


Is  this  topic  or  skill  part  of  the  actual  curriculum? 

Use  teacher  questionnaires  as  focus  for  discussion 


Y 


Q5 


What  are  the  other  factors  affecting  performance? 

Use  teacher/student  questionnaires  as  focus  for  discussion 

•    Study  state  report 


V 


Form  committees 

Set  time  lines 

Give  progress  reports 
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Question  1:  What  are  the  strengths  and  weaknesses  of  our  instructional  program? 

Scores  for  the  subcategories  within  each  subject  area  are  reported  on  pages 
10  through  13  of  the  report.  The  number  of  items  used  is  also  listed.  Examples 
of  the  actual  items  are  given  in  the  Description  of  Test  Content  and  Reporting 
Categories,  which  should  be  in  your  school  but  is  also  available  from  the  Department 
of  Education. 

The  first  step  in  evaluating  your  program  is  to  examine  your  performance 
on  the  various  components  of  the  assessment.  Within  most  subject  areas,  the 
assessment  material  has  been  classified  according  to  content  and  to  the  type  of 
thinking  required.  In  the  area  of  Reading,  the  categories  reflect  components  of 
reading  and  the  kind  of  passage  on  which  the  questions  were  based.  Whatever 
your  total  score,  it  is  important  to  start  your  evaluation  by  examining  the  individual 
categories. 


Educational 

Assessment 
Program 
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In  the  visual  display,  the  diamond  indicates  the  score  which  your  students 
received,  while  the  bar  indicates  the  relative  precision  of  that  score.  You  will  notice 
that  some  bars  are  longer  than  others.  This  is  a  function  of  three  factors:  the 
number  of  students  in  your  school  who  took  the  tests,  the  number  of  questions 
in  that  category,  and  the  percentage  of  students  answering  each  item.  It  is  an 
acknowledgment  of  the  fact  that  scores  achieved  on  a  test  are  only  estimates  of 
performance  and  that  some  scores  are  more  precise  than  others.  As  a  consequence, 
the  general  rule  in  comparing  scores  is  to  pay  particular  attention  only  to  those 
scores  which  do  not  overlap  the  bars. 
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The  simplest  way  to  identify  areas  of  weakness  in  each  subject  area  is  to  use 
the  total  score  as  a  reference  and  draw  a  vertical  line  from  that  point.  You  can 
then  note  how  the  bar  for  each  subskill  lies  in  relation  to  that  line.  Bars  that 
fall  to  the  right  of  the  line  represent  relative  strengths,  while  those  that  fall  to 
the  left  signify  relative  weaknesses. 


Qustion  2:  Are  we  satisfied  that  this  is  an  area  of  weakness? 

•  Do  other  test  scores  verify  the  assessment  scores? 

•  Do  teacher  judgments  confirm  results? 

•  Is  there  other  evidence  to  confirm  results? 

Whether  a  subcategory  can  be  considered  an  area  of  weakness  is  a  matter 
of  judgment.  That  judgment  rests  on  a  number  of  factors  which  should  be 
considered  by  your  curriculum  area  experts  and  the  teachers  involved. 

For  most  schools  the  assessment  results  are  one  of  many  sources  of  information 
on  school  performance.  Commercial  standardized  test  results  are  available,  as  are 
scores  on  department-wide  tests  and  classroom  measures.  The  extent  to  which 
information  obtained  from  these  other  sources  converges  is  a  measure  of  its  validity. 

On  the  other  hand,  the  assessment  may  point  to  weaknesses  that  exist  but 
are  not  measured  by  other  instruments.  The  assessment  was  designed  to  cover 
a  broad  range  of  knowledge  and  types  of  thinking  that  are  representative  of  each 
subject  area.  As  a  result  it  measures,  along  with  more  traditional  skills,  certain 
strategies  and  kinds  of  thinking  that,  although  considered  important  by  content 
area  experts,  are  not  explicitly  taught  or  tested  in  some  schools.  For  example,  the 
metacognitive  skills  that  are  included  in  our  reading  framework  are  a  departure 
from  standardized  reading  tests.  Although  this  particular  innovation  was  designed 
to  make  the  assessment  results  more  relevant  to  instruction  and  more  in  keeping 
with  current  research  on  the  reading  process,  achievement  in  this  area  may  not 
be  congruent  with  scores  obtained  on  more  traditional  reading  tests.  This  lack 
of  congruence  is  an  important  consideration  in  interpreting  test  results.  Unless 
a  test  is  aligned  with  the  instructional  program,  it  will  not  be  sensitive  to  effects 
of  that  program. 

Curriculum  evaluation  is  not  a  mechanical  process  that  can  be  followed 
through  a  series  of  check-lists.  Basically/  it  is  a  matter  of  examining  the  evidence 
and  making  informed  judgments.  For  this  reason,  when  choosing  an  area  for 
curriculum  improvement,  it  is  important  to  examine  the  various  sources  of 
information  carefully  before  deciding  where  resources  should  be  placed. 
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Question  3:  Is  this  topic  or  skill  part  of  our  intended  curriculum? 

•  Does  this  topic  or  skill  form  part  of  our  written  objectives? 

•  Is  it  covered  adequately  in  the  textbooks  and  curriculum  guides? 

•  7s  instruction  supported  by  currriculum  materials  and  other  learning  aids? 

Recently,  educators  have  defined  three  different  aspects  of  the  word 
"curriculum."  They  refer  to  the  written  curriculum — that  which  is  contained  in 
the  goals,  objectives,  and  scope  and  sequence  charts  in  our  schools — as  the  intended 
curriculum.  They  note  that  this  intended  curriculum  is  distinct  from  the  actual 
curriculum — that  which  is  filtered  through  the  teacher  and  is  taught  in  the 
classroom  itself.  Finally,  there  is  the  achieved  curriculum,  what  is  learned  by 
students  and  usually  reflected  on  test  results.  In  order  to  understand  why  a 
particular  curriculum  has  or  has  not  been  achieved,  it  is  necessary  to  examine  how 
the  first  two  terms  are  understood  and  employed. 

Although  a  written  curriculum  exists  in  many  schools,  the  assessment  offers 
an  opportunity  for  school  staff  to  come  together  to  review  and  revise,  if  necessary. 
Using  the  descriptions  contained  in  the  Description  of  Test  Content  and  Reporting 
Categories  and  the  appropriate  sample  items  as  guides,  the  committee  should 
consider  the  material  covered  in  their  school(s)  through  their  curriculum  objectives 
and  instructional  materials.  The  subcategories  in  each  subject  area  represent 
different  types  of  knowledge,  skills  or  kinds  of  thinking.  Committees  should  look 
at  these  categories  and  ask: 

Does  this  content  or  type  of  thinking  match  our  current  list  of  curriculum 
objectives?  Should  it?  If  so,  by  what  grade  level?  For  all  students? 

The  intended  curriculum  is  not  only  reflected  in  a  set  of  written  objectives. 
To  be  meaningful,  it  must  also  be  supported  by  textbooks,  curriculum  materials 
and  other  learning  aids.  At  this  stage,  when  examining  the  intended  curriculum, 
it  is  important  that  committees  go  beyond  what  it  written  to  what  actually  exists 
for  teachers  to  use  and  how  it  is  interpreted  in  learning  materials. 

Each  school  must  decide  for  itself  what  aspects  of  the  subject  are  most  relevant 
and  beneficial  for  its  students.  This  will  depend  not  only  upon  the  characteristics 
and  previous  experiences  of  the  students,  but  also  upon  the  particular  emphases 
of  your  curriculum  and  the  resources  available.  Remember  that  the  assessment 
was  designed  to  include  all  the  skills  and  concepts  that  are  considered  appropriate 
in  the  subject  area  tested.  Given  the  broad  scope  of  the  assessment,  it  is  highly 
unlikely  that  any  school  would  be  able  to  cover  thoroughly  every  subcategory, 
nor  would  such  an  endeavor  be  productive  given  the  time  available. 

On  the  other  hand,  it  is  important  to  consider  your  relative  performance  on 
the  different  subskills.  Very  low  scores  can  be  flags  for  attention.  Since  the 
assessment  includes  very  easy  as  well  as  very  difficult  items  within  each  subskill, 
a  low  score  suggests  that  your  students  may  be  missing  out  on  fundamental 
concepts  within  the  subject  area  as  a  whole. 
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Question  4:  Is  this  topic  or  skill  a  part  of  the  actual  curriculum? 

•  Is  this  skill  or  topic  actually  being  taught  in  the  classroom?    To  what  level  of 
mastery? 

•  Are  there  adequate  resources? 

•  Are  teachers  interested  and  supportive  of  this  curriculum? 

•  Is  there  the  support  for  teachers,  including  in-service? 

If  the  skill  involved  is  one  that  has  formed  part  of  your  intended  curriculum 
and  remains  an  area  of  weakness,  it  is  important  to  examine  the  actual  or  delivered 
curriculum.  In  other  words,  is  the  skill  being  taught  in  the  classroom  and  how 
is  it  being  taught? 

There  are  many  reasons  why  content  areas  or  skills  are  not  taught  or  are  taught 
sketchily.  These  include  lack  of  time,  lack  of  supporting  materials,  lack  of  interest 
on  the  part  of  the  teacher  or  students,  and  lack  of  confidence  by  the  teacher  that 
s/he  can  teach  it  effectively.  Each  reason  carries  its  own  solution.  For  example, 
teachers  may  believe  that  the  topic  is  inappropriate  for  their  students  or  a  portion 
of  their  students.  They  may  believe  that  the  topic  is  untimely,  or  presupposes 
experiences  that  children  have  not  yet  attained.  On  the  other  hand,  the  skill  or 
topic  may  not  be  taught  bcause  of  lack  of  resources.  Books  or  appropriate  materials 
may  not  be  available.  Teachers  themselves  may  not  have  sufficient  training  to 
feel  competent  to  teach  it  to  the  degree  that  ensures  learning.  To  acknowledge 
why  a  skill  or  topic  is  not  being  taught  is  as  important  as  discovering  whether 
it  is  being  taught. 

The  teachers'  questionnaire  should  be  a  valuable  source  for  promoting 
discussion.  We  have  encouraged  schools  to  keep  those  questionnaires  (returning 
only  the  answer  sheets)  so  that  they  could  be  used,  with  the  state  summary  report, 
to  structure  discussion  on  instructional  methods  and  the  actual  content  of 
instruction.  At  the  fourth  and  eighth  grade  levels,  these  questionnaires  contain 
actual  items  which  were  used  on  the  assessment.  It  will  be  useful  to  confirm 
if  these  items  reflect  the  kind  of  knowledge  that  you  expect  your  students  to  know 
and  understand  and  to  what  extent  consensus  exists  among  teachers  of  different 
classes. 

In  a  small  setting  in  which  thoughts  and  feelings  can  be  expressed  freely, 
it  may  be  possible  for  the  group  to  discuss  each  of  the  objectives  in  turn.  In 
other  settings,  it  may  be  more  useful  to  conduct  a  simple  written  survey,  asking 
each  teacher  to  indicate  to  what  extent  (mastery,  introductory,  or  reviewed)  each 
of  the  objectives  is  taught  in  each  of  his  or  her  classes.  In  any  event,  keep  your 
discussions  frank  and  nonjudgmental.  Do  not  set  out  with  the  expectation  that 
every  subskill  in  the  assessment  (or  even  in  your  curriculum  guide)  can  be  taught 
to  mastery  by  every  teacher.  Instead,  keep  track  of  the  reasons  why  objectives 
are  not  being  taught.  Consider  them  as  problems  to  be  solved,  not  as  "excuses" 
for  a  job  not  done. 
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Question  5:  What  are  the  other  factors  affecting  performance? 

If  the  objective  is  in  both  the  intended  and  the  actual  curriculum,  but  students 
are  performing  poorly  in  relation  to  their  performance  in  the  subject  as  a  whole, 
you  must  ask  another  set  of  questions. 

•  Do  our  students  understand  the  concepts  as  well  as  we  believe  they  do? 

The  assessment  was  designed  to  reach  beyond  factual  knowledge  and  to  tap 
students'  understanding  of  concepts  and  their  ability  to  apply  those  concepts  to 
familiar  and  unfamiliar  situations.  In  many  instances,  the  items  differ  from  those 
questions  found  in  textbooks  at  the  end  of  chapters,  and  neatly  tied  in  with  the 
previous  instruction.  Students  who  are  adept  at  answering  the  latter  are  often 
unprepared  for  more  challenging  questions  in  different  contexts.  The  State  Summary 
Report  discusses  some  of  the  discrepancies  that  occur  between  students'  knowledge 
of  facts  and  their  performance  when  asked  to  use  those  facts  in  solving  problems. 

Classroom  discussion  can  also  give  false  information  when  teachers 
consistently  ask  for  correct  responses.  For  example,  it  has  been  observed  that 
students'  lack  of  understanding  of  mathematical  and  scientific  terms  is  often 
masked  by  their  ability  to  supply  the  correct  term  when  asked.  Furthermore,  in 
classroom  discussions,  if  teachers  call  predominantly  on  volunteers  who  know  the 
answer,  they  may  get  an  inflated  impression  of  how  the  class  is  doing  as  a  whole. 
Some  close  questioning  of  a  wider  group  of  students  may  give  a  more 
representative  assessment  of  progress. 

At  the  eighth  and  twelfth  grade  levels,  students  were  asked  to  give  a  general 
indication  of  the  grades  that  they  receive  at  school.  The  assessment  scores  for 
students  responding  to  each  category  are  given  for  school  (when  the  number  of 
students  is  greater  than  20),  district  and  state.  This  comparison  will  give  you 
some  indication  of  possible  grade  inflation  (or  deflation)  among  the  teachers  in 
your  own  school.  Although  some  students  may  exaggerate  their  grades,  it  has 
been  found  that  they  are  generally  quite  accurate  in  their  reports,  and  the  overall 
average  response  is  a  reliable  indication  of  grade  achievement. 

In  addition,  the  results  from  the  open-ended  sections  of  the  assessment  may 
suggest  ways  teachers  might  use  open-ended  questions  to  assess  their  students' 
understanding  of  different  topics.  Copies  of  the  reports  on  these  results  will  be 
sent  to  all  schools  early  in  1989. 

•  Are  our  expectations  too  low? 

Students  may  be  capable  of  more  than  we  demand  of  them.  For  example, 
studies  have  shown  (and  our  results  have  confirmed)  the  strong  influence  of  prior 
knowledge  in  many  "thinking"  tasks.  When  students  have  knowledge  of  the 
content  area,  they  are  able  to  interpret  visual  data  and  to  answer  comprehension 
questions  that  they  find  difficult  in  other  contexts.  In  other  cases,  students  often 
use  informal  knowledge  to  respond  correctly  to  problems  in  areas  not  covered 
in  the  curriculum.  This  suggests  that  expectations  of  students'  performance  may 
limit  what  we  allow  them  to  learn. 
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•  Is  learning  articulated  throughout  the  grades  and  reinforced  across  subject  areas? 

Although  the  assessment  takes  place  at  a  particular  grade  level,  it  measures 
all  the  learning  that  has  taken  place  up  to  that  time.  Students  may  forget 
unconnected  facts,  but  we  would  expect  that  their  basic  understanding  of  the  subject 
would  continue  to  be  applied  to  more  recent  instruction.  On  the  other  hand,  one 
of  the  most  important  findings  of  the  assessment  is  the  lack  of  reinforcement  of 
learning  across  subject  areas,  despite  general  themes  that  permeated  all  subjects. 
The  extent  to  which  subjects  are  interrelated  throughout  the  curriculum  is  discussed 
in  the  State  Summary  Report  where  specific  examples  are  given.  Schools  should 
study  this  report  and  examine  their  own  instruction  for  evidence  that  writing  is 
not  isolated  to  language  arts,  graphic  interpretation  is  not  the  sole  province  of 
the  science  teacher  and  reading  is  taught  across  content  areas. 

•  Are  we  attempting  to  reach  beyond  school  gates  to  involve  parents  and  others  in 
promoting  our  objectives? 

The  student  questionnaire  results  give  evidence  for  the  strong  association 
between  out-of-school  activities  and  performance  on  the  assessment.  Average 
assessment  scores  are  given  at  state,  district  and  school  levels  for  those  students 
who  responded  in  different  ways  to  questions  regarding  free  time  and  work 
activities.  The  percentage  of  students  in  your  school  who  responded  to  the  various 
options  is  also  given  to  help  you  compare  the  activities  of  your  students  with 
others. 

Despite  the  fact  that  the  school  has  a  limited  role  in  affecting  student  behavior 
outside  of  its  walls,  it  can  encourage  the  reinforcement  of  learning  at  home.  The 
strong  effect  of  social  factors  such  as  parental  education  may  be  due,  in  large 
measure,  to  this  kind  of  reinforcement.  For  example,  the  statewide  summary 
report  shows  that  certain  student  activities  are  associated  with  performance.  That 
these  activities  (e.g.,  reading  for  pleasure,  television  watching,  etc.)  are  related 
to  achievement  does  not  mean  that  they  cause  achievement.  Nevertheless,  students 
who  spend  their  time  outside  of  school  applying  their  instruction  in  different 
contexts  are  essentially  extending  and  reinforcing  what  they  have  learned  during 
the  school  day. 

In  addition,  schools  have  the  opportunity  of  bringing  the  outside  world  into 
the  school.  Class  trips,  activities  based  on  practical  and  meaningful  problems,  guest 
speakers — all  contribute  to  understanding  the  relevance  of  school  learning  to 
everyday  life. 


Final  Steps:  Preparing  an  action  plan 

•  Does  the  plan  document  the  most  important  weaknesses?  strengths? 

•  Is  the  plan  based  on  thorough  diagnosis  from  credible  evidence? 

•  Are  steps/recommendations/plans  for  proceeding  clearly  specified? 
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•  Are  resources,  participation  and  time  delineated?  realistic? 

•  Is  an  evaluation  plan  for  determining  the  effectiveness  of  the  proposed  plan  detailed? 

If  your  plan  is  to  be  effective,  there  are  some  important  points  to  remember. 
One  is  the  need  for  enlisting  the  support  of  all  involved.  To  the  greatest  extent 
possible,  order  your  priorities  on  the  basis  of  consensus.  Unless  there  is  general 
interest  and  enthusiasm  for  the  effort,  little  will  be  accomplished.  Secondly,  it 
is  important  to  be  realistic  in  terms  of  the  resources  and  human  effort  that  will 
be  involved.  Before  mounting  any  effort,  review  your  textbooks  and  available 
materials  to  make  sure  that  you  have  the  resources  to  teach  the  skills  and  content 
that  you  have  chosen.  Be  prepared  to  spend  more  time  and  energy  than  anticipated 
on  instituting  the  new  areas  of  instruction.  Finally,  don't  concentrate  on  a  single 
year.  Skills  and  knowledge  are  cumulative.  It  is  important  that  curriculum 
objectives,  however  specific,  be  seen  as  the  logical  extension  of  earlier  learning, 
to  be  reinforced  in  the  later  grades. 

In  a  complex  process  such  as  described  here,  it  is  sometimes  possible  to  lose 
the  way.  Groups  become  distracted;  other  commitments  take  precedence.  The 
impact  of  the  assessment  results  begins  to  recede.  For  these  and  other  reasons, 
it  is  important  to  do  two  things:  set  a  time  line  and  plan  a  report. 

Setting  realistic  deadlines  for  stages  in  your  evaluation  and  for  performance 
goals  in  your  action  plan  will  strengthen  your  commitment  to  the  project.  A  general 
time  line  can  be  anticipated,  with  deadlines  for  specific  steps,  such  as  teacher 
surveys  or  reviews  of  textbooks,  to  be  implemented  along  the  way.  The  fact  that 
you  have  a  sequential  plan  and  regard  it  seriously  will  help  participants  feel  a 
sense  of  progress. 

Secondly,  regular  progress  reports  to  faculty  and  parents  will  help  focus 
attention  on  the  overriding  questions  of  "what  have  we  learned  in  our  evaluation?" 
and  "what  are  we  doing  about  it?"  Reports  should  address  perceptions  and 
problems,  as  well  as  more  tangible  results.  They  will  form  an  important  public 
record  of  your  progress  in  school  improvement. 
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CHAPTER  FIVE 
CASE  HISTORIES 

In  the  spring  of  1988,  the  Department  surveyed  all  schools  that  had  taken 
part  in  the  assessment,  asking  if  they  had  used  the  results  of  the  assessment  for 
an  evaluation  of  their  curriculum.  Of  the  1050  schools  which  replied,  approximately 
half  responded  affirmatively,  stating  that  they  had  used  the  results  for  review  and 
improvement.  In  more  extensive  questioning,  we  were  struck  by  the  flexibility  of 
their  approaches.  It  was  obvious  that  schools  looked  at  the  assessment  in  terms 
of  their  own  perceived  needs. 

This  chapter  describes  four  school  systems  which,  although  generally  following 
a  similar  procedure,  focused  on  different  aspects  of  the  assessment  results.  All 
have  generously  agreed  to  share  their  material  with  us  and  all  are  willing  to  discuss 
their  procedures  with  others  who  are  interested  in  embarking  on  a  similar  process. 


Tewksbury 

The  school  system  in  Tewksbury  concentrated  on  the  areas  of  mathematics 
and  reading.  Although  the  system  assumed  that  teachers  followed  the  scope  and 
sequence  chart  outlined  by  their  basal  reading  program  and  adopted  math  series, 
they  were  aware  that  the  actual  or  taught,  curriculum  often  contrasts  with  the 
intended  curriculum  of  textbooks  and  curriculum  guides.  In  other  words,  individual 
teachers  often  modify  their  instruction  according  to  what  they  perceive  to  be  the 
needs  of  their  own  pupils.  For  effective  changes  to  be  made,  it  was  important 
to  know  what  was  actually  taught,  to  what  extent,  and  to  whom. 

Since  the  assessment  reflects  cumulative  learning  rather  than  the  curriculum 
in  any  particular  grade,  detailed  information  was  sought  from  all  teachers  in  the 
system.  Large  reporting  categories  were  broken  down  to  the  finer  objectives  that 
were  delineated  in  the  Department's  1986  publication,  Description  of  Test  Content 
and  Reporting  Categories.  These  were  circulated  to  grade  1  through  12  teachers,  who 
were  asked    to  supply  the  following  information  for  each  objective: 

•  the  total  amount  of  time  spent  in  instruction 

•  the  level   of  mastery   expected:   readiness,   introduction,   developmental, 
maintenance 

•  the  average  achievement  level  of  the  focal  class 

•  the  time  during  the  school  year  when  the  objective  was  taught. 

This  information  was  then  compiled  and  summarized  (see  example  below)  to 
be  used  as  the  basis  for  discussion  of  curriculum  change.  With  components  of 
the  actual  curriculum  laid  out  on  paper,  it  was  obvious  that,  despite  common 
textbooks  and  scope  and  sequence,  there  was  great  variability  in  the  emphasis 
given  to  the  various  topics.    Furthermore,  not  only  were  there  differences  among 
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teachers  at  the  same  grade  level,  the  amount  of  emphasis  was  not  necessarily 
developmental  throughout  the  grades.  For  example,  a  great  deal  of  time  was  spent 
on  some  skills  in  the  early  grades,  then  not  touched  upon  until  later  grades  when 
they  were  reintroduced  at  considerable  length. 

Sample  from  the  Informational  Summary  of  Information  on  Assessment  Reporting  Categories 
by  Tewksbury  teachers. 


MEAP  OBJECTIVES  CONSENSUS  BY  GRADE 

READING 
GRADE  THREE: 

Vocabulary: 

3.1.1        Identify  synonyms  and  antonyms. 

This  is  taught  in  Grade  1  for  five  half-hour  blocks  using  the  process 
of  Readiness.  The  medium  used  is  text  and  the  class  skill  is  average.  It 
is  taught  in  the  third  term. 

This  is  taught  in  Grade  2  for  five  half-hour  blocks  using  the  process 
of  Development.  The  medium  used  is  text  and  the  class  skill  is  average. 
It  is  taught  in  the  second  and  third  term. 

This  is  taught  in  Grade  3  for  three  half-hour  blocks  using  the  process 
of  Maintenance.  The  medium  used  is  text  and  the  class  skill  is  average. 
It  is  taught  in  the  third  term. 


3.1.2       Determine  the  meaning  of  a  word  that  has  multiple  meanings, 
depending  on  its  use  in  a  sentence. 

This  is  taught  in  Grade  1  for  five  half-hour  blocks  using  the  process 
of  Development.  The  medium  used  is  text  and  the  class  skill  is  average. 
It  is  taught  in  the  third  term. 

This  is  taught  in  Grade  2  for  nine  half-hour  blocks  using  the  process 
of  Development.  The  medium  used  is  text  and  the  class  skill  is  average. 
It  is  taught  in  the  second  and  third  term 


The  summary  of  information  not  only  gave  teachers  important  background  on  the 
variability  in  instruction,  but  it  formed  a  basis  for  curriculum  revision.  Teachers 
used  it  as  a  discussion  paper  as  they  met  to  form  a  consensus  on  instructional 
time  and  objectives.  Together,  they  considered  such  questions  as: 

•  How  many  blocks  of  time  should  be  devoted  to  this  objective? 

•  How  does  it  fit  into  the  instructional  sequence? 

•  At  what  level  should  we  expect  mastery  for  the  average  child? 
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Again,  the  actual  curriculum  as  reflected  in  the  informational  summary  was 
compared  not  only  with  the  scope  and  sequence  of  the  intended  curriculum  but 
also  with  the  consensus  curriculum  that  resulted  from  their  workshops. 

According  to  Richard  Griffin,  principal  of  Tewksbury  Junior  High  School: 

This  analysis  during  our  released-time  workshops  should  prove  helpful  in  two  ways.  First, 
it  should  lead  to  improved  emphasis  on  these  objectives  in  current  teaching;  secondly, 
it  will  allow  for  a  comparison  when  the  next  set  of  test  results  are  released  next  fall 
We  should  be  able  to  tell  if  this  emphasis  is  related  to  teaching  and  eventually  shows 
in  test  results.  At  least  we  can  now  distinguish  our  actual  curriculum  from  an  idealistic 
textbook  publisher's  intended  curriculum. 

Dr  Edward  Farley,    Assistant  Superintendent. 

Richard  Griffin,  Principal 
Tewksbury  Junior  High 

Millsbury 

Millbury  had  been  satisfied  with  their  school  achievement.  On  the 
standardized  achievement  tests  that  were  routinely  administered,  students 
performed  reasonably  well.  However,  when  the  assessment  scores  were  reported, 
Millbury  was  surprised  at  their  results,  particularly  at  the  high  school  level.  They 
wanted  to  determine  the  reasons  for  the  discrepancy  between  the  two  sets  of  test 
results. 

They  began  with  the  program  information  that  had  been  reported  in  the 
student  questionnaires.  Comparing  their  eleventh  grade  students  to  those  in  similar 
districts,  they  noted  the  disproportionately  large  number  who  were  in  general 
rather  than  academic  programs.  This  led  to  the  following  questions: 

1.  Are  students'  perceptions  of  their  school  program  realistic?  (i.e.,  Are  two  thirds 
of  eleventh  graders  actually  in  general  programs  and  one  third  in  college 
preparatory?) 

2.  If  this  perception  is  accurate,  why  are  so  few  students  selecting  an  academic/ 
college  preparatory  program  compared  to  comparable  towns? 

3.  Could  there  be  other  factors  (in  addition  to  expectations  and  course  offerings) 
affecting  the  small  number  of  students  selecting  a  college  preparatory  program? 

4.  What  is  the  future  impact  of  so  few  students  selecting  an  academic /college 
preparatory  program? 

These  questions,  as  well  as  more  general  ones,  led  to  the  formation  of  a  task 
force  to  examine  issues  that  might  affect  performance.  This  committee  was 
composed  of  parents,  administrators,  and  teachers.  It  began  its  investigation  by 
inviting  various  groups  to  appear  before  it.  Asking  "What  are  the  issues  as  you 
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see  them?  What  are  the  strengths  of  the  system?  What  are  its  weaknesses?",  the 
committee's  inquiry  soon  expanded  beyond  the  eleventh  grade  assessment  results 
to  consider  the  entire  system. 

With  an  initial  deadline  of  the  spring  of  1987,  the  committee's  work  resulted 
in  the  following  recommendations  which  initiated  major  changes  in  the  Millbury 
school  system. 

Recommendation  1.  A  more  effective  and  continuous  evaluation  of  student 
progress 

Result:  A  more  rigorous  standarized  achievement  test  was  adopted  and  the  district 
began  testing  at  every  grade  level.  The  results  of  these  tests,  as  well  as  the  Basic 
Skills  and  assessment  tests,  are  used  to  evaluate  curriculum  on  an  annual  basis. 

Recommendation  2.    The  intergration  of  instruction  at  all  grade  levels 

Result:  A  system-wide  curriculum  coordinator  was  hired  whose  major  responsibility 
was  to  establish  articulation  among  the  various  levels. 

Recommendation  3.    Greater  involvement  of  parents  in  school  policy 

Result:  A  system-wide  homework  policy  was  formulated  and  expectation  sheets 
were  initiated.  These  expectation  sheets,  which  outline  the  standards  for  each 
course,  are  now  distributed  to  all  parents  to  be  signed  and  returned  to  the  school. 
Essentially,  they  form  a  contract  between  home  and  school  on  the  terms  of 
instruction. 

Recommendation  4.  Stronger  control  over  the  sequence  of  courses  that  students 
take 

Result:  Millbury  has  increased  graduation  requirements,  including  three  years  of 
science,  social  studies  and  mathematics,  to  insure  that  students  do  not  abandon 
academic  courses  in  their  junior  year.  In  addition,  there  is  a  continuing  effort  to 
increase  dialogue  between  guidance  staff  and  student  and  parents. 

Recommendation  5.  A  systematic  approach  to  curriculum  development  and 
review 

Result:  An  annual  process  of  departmental  goal-setting  and  follow-up  has  increased 
accountability  in  each  of  the  learning  areas.  New  programs  in  reading  and  science 
have  been  introduced  in  grades  K  through  6,  and  each  learning  area  has  begun 
the  process  of  developing  scope  and  sequence  charts  throughout  the  grades. 

Recommendation  6.  Staff  teachers  should  have  a  greater  role  in  curriculum 
development 

Result:  Teachers  are  now  involved  in  systemwide  committees  on  curriculum,  staff 
development  and  testing. 

Dr  Alfred  Sylvia,    Superintendent 

Dr.  David  Troughton,    Curriculum  Coordinator 
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Pittsfield 

Pittsfield  is  fortunate  in  having  curriculum  directors  in  each  of  the  subject  areas 
covered  by  the  1986  assessment.  After  receiving  the  school  and  district  reports, 
each  of  these  directors  analyzed  the  districtwide  results  across  all  grade  levels 
and  reported  their  instructional  implications  to  each  school  building  principal. 
Principals,  in  turn,  met  with  their  own  staff  to  relate  these  reports  to  their  own 
school  population  and  to  review  the  need  for  resources,  staff  training  and 
instructional  change. 

Students  in  the  Crosby  School  had  performed  relatively  well,  scoring  close 
to  the  state  average  and  at  the  top  of  their  Comparison  Score  Band.  However, 
the  principal  and  his  staff  noted  anomalies  in  some  curriculum  areas  that  were 
cause  for  concern.  For  example,  although  the  school's  average  score  in  mathematics 
was  satisfactory,  the  Curriculum  Area  Results  reflected  large  variations  in 
performance.  While  the  school's  score  in  mathematical  computation  was  very 
high,  scores  in  numeration  and  problem  solving  were  significantly  lower  with 
differences  of  160  and  100  points.  The  size  of  the  differential  suggested  that  the 
curriculum  was  too  narrowly  focused  on  skills  at  the  expense  of  conceptual 
understanding.  It  indicated  that  students  were  weak  in  their  understanding  of 
mathematical  concepts  and  in  recognizing  the  underlying  numerical  operations 
demanded  in  verbal  problems. 

On  the  basis  of  this  analysis,  the  staff  decided  to  supplement  the  school's  basic 
curriculum  in  mathematics  by  encouraging  more  active  involvement  in 
mathematical  activity.  Manipu  Stives  were  purchased  for  use  in  the  early  grades. 
Work  with  older  students  focused  on  the  use  of  language  in  word  problems.  In 
general,  less  reliance  was  to  be  placed  on  mimeographed  worksheets,  and  more 
on  student  activity. 

Crosby  School  also  focused  on  the  results  of  the  student  questionnaires,  noting 
that  the  more  time  students  spent  in  reading  for  their  own  pleasure,  the  higher 
their  score  on  the  assessment.  The  state  summary  reported  that  53%  of  third  grade 
students  replied  that  they  spent  no  time  in  voluntary  reading.  However,  for  those 
who  read  as  little  as  a  half-hour  a  day,  reading  scores  showed  a  dramatic  increase 
and  continued  to  show  incremental  improvement  up  to  2  hours  a  day. 

This  information  was  used  to  involve  parent  groups  in  efforts  to  encourage 
independent  reading  among  their  children.  Reading  is  Fundamental,  an  existing 
federally  based  reading  program  designed  to  encourage  reading  in  grades  K-2, 
was  expanded  through  grade  5.  Funds  were  obtained  through  the  PTO  and  federal 
sources  for  the  free  distribution  of  paperbacks  to  these  grades  and  activities  were 
focused  on  teaching  parents  how  to  motivate  children  to  read  independently.  As 
a  result,  there  was  renewed  emphasis  on  the  reinforcement  of  school  learning 
through  the  involvement  of  parents  in  the  learning  process. 

Dr.  William  Travis,    Curriculum  Director 

Eugene  O'Brien,  Principal 

Highland  Elementary  School    (formerly  of  the  Crosby  School,  Pittsfield) 

24 


Freetown-Lakeville 

The  district  Freetown-Lakeville  consists  of  two  independent  K-3  schools,  a 
district  junior  high  school  and  a  district  high  school.  When  Superintendent  Nancy 
Sullivan  reviewed  the  assessment  results  she  was  intent  on  creating  improved 
communication  and  curriculum  integration  among  these  four  separate  units. 

Her  first  step  was  to  create  a  council  of  school  administrators  to  review  the 
assessment  results  and  propose  a  plan  of  action.     Among  its  goals  were: 

•  the  creation  of  a  K-12  focus  for  curriculum  articulation  that  would  build 
from  year  to  year; 

•  improved  curriculum  management; 

•  the  development  of  curriculum  guides; 

•  the  involvement  of  90%  of  staff  in  curriculum  review  and  decisions; 

•  improved  assessment  results. 

When  Dr.  Sullivan  presented  the  proposal  to  the  four  school  committees 
involved,  they  endorsed  it  unanimously,  and  work  was  begun  in  January  1987 
on  the  review  and  improvement  of  what  and  how  students  learned  throughout 
the  district. 

In  addition  to  the  administrative  council,  the  process  involved  two  types  of 
committees  that  ensured  the  involvement  of  all  staff.  One  was  a  K-12  Steering 
Committee  which  was  primarily  concerned  with  the  coordination  of  effort  and  the 
overall  integration  of  curriculum  and  instruction  throughout  the  system.  The 
second  type  were  school  curriculum  committees,  including  all  teachers,  which 
looked  at  specific  subject  areas  and  made  recommendations  about  content  and 
resources.  Together,  these  committees  ensured  that  the  resulting  recommendations 
would  be  specific  enough  for  the  classroom  teacher  and,  at  the  same  time,  would 
be  consonant  with  system-wide  educational  goals. 

In  March,  all  professional  faculty  and  staff  in  the  four  schools  met  as  a  group 
to  review  the  mathematics,  reading  and  science  curricula  as  they  related  to  the 
Massachusetts  assessment  program.  The  goals  for  this  initial  meeting  were  to  make 
decisions  about  changes,  to  plan  development,  and  to  develop  the  concept  of  K- 
12  articulation.  Focusing  on  the  three  grade  levels  tested  by  the  assessment,  school 
faculty  examined  each  of  the  specific  objectives  in  light  of  the  existing  curricula, 
recording  when  it  was  introduced  into  the  curriculum  and  the  degree  of  emphasis 
that  was  accorded  to  it.  (See  Form  A,  following.)  On  the  basis  of  these  discussions, 
teachers  made  prelimary  recommendations  in  regard  to  possible  changes  in 
inclusion  and  emphasis.  At  this  time,  they  also  noted  implications  for  resources, 
such  as  more  material  or  more  training. 
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The  results  of  these  meetings  were  consolidated  and  sent  back  to  each  building. 
After  subject  area  subcommittees  had  reviewed  and  discussed  these  results,  a 
second  round  of  meetings  was  then  held  to  formulate  final  recommendations.  These 
meetings  focused  on  the  subject  areas.  Each  objective  was  again  examined  and 
decisions  were  made  about  whether  it  should  be  placed  in  the  curriculum,  the 
amount  of  emphasis  it  should  receive,  and  the  training  and  resources  needed  for 
instructional  implementation. 


Final  M.E.A.P.  Recommendations 


School: 


Check  One: 


Reading 


Mathematics 


Science 


For  elementary  school,  record  grade  level  here 

For  middle  school,  record  ability  level  here  

For  high  school,  record:        Course  name 


&  grade  level 


Grades  taught  at 
Ability  levels 


Identify  reporting  category:. 
Identify  Objective: 


To  improve  mastery  at  the  appropriate  grade  level,  either  grade  3,  7,  or  11,  we  recommend 
the  following: 

In  our  curriculum? 

Place  in  curriculum/does  not  exist 

Should  be  taught/in  curriculum  but  not  taught 

Put  in  writing/is  taught  but  not  in  writing 

Degree  of  emphasis? 

Should  be  introduced  at  this  grade  level 

Should  be  mastered  at  this  grade  level 

Should  be  reviewed  at  this  grade  level 

Enough  class  time? 

More  class  time  should  be  devoted  to  this  objective 

Enough  practice  time? 

Students  should  be  assigned  more  practice  time 

Proper  materials? 

We  need  different  resource  materials  than  now  exist 

We  need  more  of  the  same  resource  materials  than  now  exist 

We  need  resource  materials/do  not  have  any 

Proper  training? 

We  need  in-service  education  in  this  area  to  be  more  effective 

Other  comments: 
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The  final  round  of  meetings,  which  included  teachers  at  all  grade  levels,  aimed 
at  a  consensus  on  what  should  be  taught  at  each  grade  level.  Objectives  and  recom- 
mendations were  again  reviewed,  this  time  in  regard  to  their  implications  for  total 
instructional  time,  instructional  material  and  availability  of  teaching  units.  Nego- 
tiation between  the  different  grade  level  and  subject  area  teachers  predominated 
these  discussions  as  the  entire  instructional  program  was  viewed  as  a  whole.  Units 
that  reflected  topics  that  were  already  part  of  the  written  objectives  were  reviewed 
for  the  degree  of  mastery  at  each  grade  level  and  the  time  required. 


M.E.A.P.  Teaching   Unit  To  Be  Emphasized/ 

Already  in  Curriculum  (in  Writing) 


School: 


Subject: . 


Faculty  member(s): 


M.E.A.P.   Category: 


M.E.A.P  Objective:    Number  # 

Recommendations: 

1.    For  grade  level  


&  Letter 


2.    Please  check  one: 


3.    Please  check  one: 


4.    Other: 


I  =  Introduce 

M  =  Master 

R  =  Review 

More  class  time  required 

More  practice  time  required 
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The  specific  resources  needed  to  implement  instruction  were  noted  and  sent 
to  each  principal  for  budget  allocations. 


M.E.A.P.  Resources  Needed  to  Implement 
M.E.A.P.  Categories  &  Objectives 


School: 


Subject:. 


Faculty  Member(s):. 


M.E.A.P.   Category:. 


M.E.A.P.  Objective:    Number  # 


&  Letter 


Resources  Required  to  Implement  Objective 
Please     Order 


Reswrct  TUte 

To  Order  From 

Quantity 

Unit  Price 

Total   Price 
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Finally,  in  cases  where  there  was  no  written  teaching  unit  to  match  the 
objective,  in-service  time,  including  paid  summer  institutes,  was  provided  for 
teachers  to  write  such  a  unit.  Among  the  major  achievements  was  in  the  area  of 
elementary  science.  Because  of  the  lack  of  instructional  units  in  one  elementary 
school,  a  consultant  and  five  teachers  joined  together  to  write  a  complete  new 
curriculum  for  elementary  science,  some  units  of  which  were  adopted  by  the  other 
elementary  school. 

As  a  result  of  Freetown-Lakeville's  extraordinary  effort,  both  intended  and 
actual  curricula  was  changed  throughout  the  grades.  It  was  a  major  effort,  fully 
supported  by  each  of  the  parties  involved.  The  school  committee  generously 
provided  the  necessary  funds.  Administrators  provided  both  financial  and 
administrative  support  for  the  teachers  involved.  The  teachers  devoted  time  and 
energy  to  serving  on  committees,  reviewing  curriculum  and  designing  new  units 
of  instruction.  According  to  Richard  Pazasis,  Director  of  Instructional  Program, 
the  experience  has  had  many  spin-offs.  As  a  result  of  the  effort,  the  K-12  Curriculum 
Committes  are  now  a  part  of  the  educational  process.  They  continue  their  work 
by  meeting  regularly  to  go  through  the  curriculum,  review  the  existing  curriculum 
guides,  update  them  and  make  them  more  effective.  Teachers,  in  turn,  are  regularly 
given  in-service  time  for  writing  curriculum  objectives,  and  determining  their 
philosophy  and  goals  as  well  as  outcomes.  Most  important,  perhaps,  is  the 
collegiality  that  has  developed  among  the  teachers  within  and  among  the  separate 
schools.  The  process  has  tied  the  elementary  schools  together  and  is  achieving 
the  articulation  that  was  hoped  for. 

Dr.  Nana/  Sullivan,    Superintendent 

Richard  Pazasis,  Director  of  Instructional  Program 
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MASSACHUSETTS  DEPARTMENT  OF  EDUCATION 

REGIONAL  CENTERS 


Micki  Siegel  or  Marcy  Devanney 

Northwest  Regional  Center 

Berkshire  Plaza 

37  Main  Street 

North  Adams,  MA       01248 

413/664-7101 

Paul  Burnim 

Greater  Springfield  Regional 

Center 

Macek  Drive 

Chicopee,  MA      01013 

413/594-8511 

Amy  Sosman 

Northeast  Regional  Center 

Jefferson  Office  Park 

790  Turnpike  St. 

North  Andover,  MA     01845 

508/689-0164  or  617/727-0600 


Laurie  Slobody 

Central  Mass.  Regional  Center 

Beaman  Street,  Route  140 

West  Boylston,  MA       01853 

508/835-6266 

Peter  Cirioni 

Greater  Boston  Regional  Center 

75  Acton  Street 

Arlington,  MA       02174 

617/641-4870 

Pat  O'Brien 

Southeast  Regional  Center 

P.O.  Box  29 

Middleboro,  MA       02346 

508/947-1231 


Other    publications    concerning    the    Massachusetts    Educational    Assessment 
Program: 

•  Description  of  Test  Content  and  Reporting  Categories 

•  State  Summary  Report,   1988 

•  Social  Studies  Report 
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